Multiple-factor verification for vision-based systems

ABSTRACT

A system and method for interaction monitoring in a retail environment that includes executing a first monitoring system and thereby generating a first evaluation of customer selection of items; executing a second monitoring system and thereby generating a second evaluation of customer selection of items; determining monitoring alignment between the first evaluation and the second evaluation of a first customer; and triggering an action in response to the monitoring alignment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation Application of U.S. patentapplication Ser. No. 16/162,292, filed on 16 Oct. 2018, which claims thebenefit of U.S. Provisional Application No. 62/572,819, filed on 16 Oct.2017, both of which are incorporated in their entirety by thisreference.

TECHNICAL FIELD

This invention relates generally to the field of computer visionapplications, and more specifically to a new and useful system andmethod for multiple-factor verification of vision-based interactions.

BACKGROUND

Computer vision, sensor fusion, and other forms of activity monitoringhave seen some early applications to commerce and payment andearly-stage exploration for future applications. Many of existingapplications are limited in nature offering only basic functionalitysuch as error prone security systems or general environmentalstatistics. Exploration in more involved user interactions such ascheckout free shopping have been exposed to be vulnerable to adversarialattacks. In many of the current solutions, a singular approach tosensing activity in an environment is applied, which exposes the systemsto a wide range of attack surfaces through which the system may fail forlegitimate actors or be gamed by exploitative actors. Thus, there is aneed in the computer vision field to create a new and useful system andmethod for applications in multiple-factor verification of vision-basedinteractions. This invention provides such a new and useful system andmethod.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of a system of a preferredembodiment;

FIGS. 2A and 2B are a schematic representations of potential monitoringsystem options;

FIG. 3 is a flowchart representation of a method of a preferredembodiment; and

FIGS. 4-8 are schematic representations of exemplary applications ofmulti-factor monitoring to different use cases.

DESCRIPTION OF THE EMBODIMENTS

The following description of the embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.

1. Overview

The system and method for multiple-factor verification of vision-basedinteractions functions to enable computer vision (CV) drivenapplications to be verified through complimentary monitoring systems.There are a variety of nascent opportunities for CV-driven applicationsand vision-based systems in the field of commerce, retail, hospitals,logistics/operations, security, consumer applications, home tools,automotive industry, emerging human-computer interfaces, and/or otherfields. The system and method may enable enhanced performance andsecurity in many of these new applications.

Computer vision can be used in a variety of ways such as identifying orclassifying objects, tracking objects, tracking or identifying people,detecting object interactions, detecting gestures and events, and/orapplying computer vision to other applications. This is a system andmethod that can use redundant and complementary monitoring systems toverify these and other CV-driven tasks as well as complex combinationsof such monitoring approaches.

In particular, the system and method can be applied in the field ofcommerce. In particular, the system and method can be used in the fieldof retail for detecting and tracking product selection of a customer orcustomers for the intent of offering an altered checkout experience. Inone preferred use-case, the system and method may be used within asystem that offers a form of “checkout free” shopping otherwisedescribed as automatic self-checkout.

The system and method may additionally or alternatively be used within asystem that offers some form of expedited or otherwise augmentedcheckout.

Herein, automatic checkout is primarily characterized by a system ormethod that generates or maintains a virtual cart (i.e., a checkoutlist) during the shopping process of a user or users, with the objectiveof knowing the possessed or selected items for billing. The checkoutprocess can occur when a customer is in the process of leaving a store.The checkout process could alternatively occur when any suitablecondition for completing a checkout process is satisfied such as when acustomer selects a checkout option within an application. In performingan automatic checkout process, the system and method can automaticallycharge an account of a customer for the total of a shopping cart and/oralternatively automatically present the total transaction for customercompletion. Actual execution of a transaction may occur during or afterthe checkout process in the store. For example, a credit card may bebilled after the customer leaves the store.

As one example variation, the system and method can employ two or moremonitoring systems for evaluating selected items intended for purchaseby a user during a checkout session. The two systems can operatesubstantially independently to evaluate the “cart” of a customer. Acomparison of the independent evaluations can then be applied indetermining how to process or respond to one or both evaluations. Inparticular, this can be applied to the checkout process where a primarysystem is used for automatic self-checkout where a virtual cart isgenerated within the store environment. The second system for itemevaluation can then be used to determine if automatic checkout canproceed for that user or if alternative handling of the user should beused (e.g., manual checkout, or receipt checking). The two system arepreferably selected to be relatively “orthogonal” in nature, which is tosay at least a subset of the attack vectors for one monitoring systemare different from the attack vectors for the second monitoring system.In general the two monitoring systems have different monitoringresolution, capabilities, and limitations which when used parallel canbe addressed when resolving monitoring misalignment to determine if anaction needs to be taken to determining how to address the misalignment.

An environment as used herein characterizes the site where the system isinstalled and operational. The system and method can be made to work fora wide variety of environments. In a preferred implementation, theenvironment is a shopping environment such as a grocery store,convenience store, micro-commerce & unstaffed store, bulk-item store,pharmacy, bookstore, warehouse, mall, market, and/or any suitableenvironment that promotes commerce or exchange of goods or services. Anenvironment is generally the inside of a building but may additionallyor alternatively include outdoor space and/or multiple locations. Inalternate use cases, the environment can include a household, an officesetting, a school, an airport, a public/city space, and/or any suitablelocation. The environment can be a locally contained environment but mayalternatively be a distributed system with wide coverage.

Multi-factor verification of the system and method can be used indetermining if a suspected error is present in the evaluation of acustomer's cart (either because of inaccuracy of one of the monitoringsystems or because of attempted deception by the customer). Applied overmultiple shoppers and over multiple checkout events, the system andmethod may improve operation of the checkout process either throughdirecting changes in one or both monitoring systems, changes in thestore operations, changes in the products, or other suitable changes. Inone implementation, the system and method could direct worker-basedmonitoring of customers (e.g., specify when and whose carts to spotcheck). A multi-factor approach in some implementations may be used tooffer a more behavior-driven metric taking into account independentmonitoring “perspectives”, which may be used for improved userexperiences in cases where one of the monitoring systems would haveresulted in a more involved checkout process.

The system and method may alternatively be applied to other CV-drivenapplications or in specific CV-driven applications. Some embodiments ofmulti-factor verification may specifically leverage the affordances ofthe system and method described herein. For example, a security barrierembodiment of the system and method may leverage one or more forms ofmonitoring beyond basic security tag monitoring for enhanced shopliftingdetection. Such a solution may allow more ubiquitous monitoring ofproducts, allowing CV-based monitoring to be used for products thattraditionally could not be tagged or that would not be cost-effective tomonitor with a physical security tag. Such a solution could similarly,allow the CV monitoring to mitigate false-positive events (e.g., falsealarms).

Herein, this system and method will be described as being applied to theexemplary retail-related application. However, the system and method canbe applied to other fields of CV-driven applications. For example, thesystem and method can be used with autonomous vehicles for verifyingcomputer vision modeling of the physical world.

As one potential benefit, the system and method may enhance user trustthrough improving reliability of an automatic checkout system. Variousaspects of the system and method may also mitigate negative impact whena monitoring system makes an error or is unable to accurately monitoritem selection. For example, item inspection as a secondary checkperformed by a worker can be selectively directed to a small subset ofcustomers to minimize the number of inspections while addressing issuesand finding adversarial customers. Additionally, inspections can bereduced to key inspection tasks (e.g., “check for package of AAAbatteries”) for a worker to minimize the amount of time during aninspection.

As another potential benefit, the system and method may enableoperationally robust checkout technology built on lower accuracymonitoring systems. The cooperative-use of the two or more monitoringsystems may result in a performance level greater than either of the twomonitoring systems. For example, a CV-based automatic checkout systemcan be used while accuracy is operating at a first accuracy level (e.g.,80% accurate), but the effective performance can achieve an accuracylevel greater than the first accuracy level (e.g., 99.9%) by using asecondary monitoring system to detect and address issues.

As another potential benefit, some embodiments of the system and methodcan be used in training and enhancing performance of one or moremonitoring systems. For example, a CV-based automatic checkout systemcan be deployed for use alongside a supplementary monitoring system, andthat CV-based automatic checkout system can improve through trainingprovided through operation. The system and method can mitigateperformance deficiencies of any one system, but can also enable theCV-based automatic checkout system to improve with the goal ofindependently achieving desired performance.

As another potential benefit, some embodiments of the system and methodmay additionally mitigate shoplifting, system manipulation, or otherforms adversarial behavior. Customers attempting to steal, confuse, ormanipulate the system can preferably be detected. As the attack vectorsof the system are minimized, “bad behavior” does not have to beaddressed harshly and the user experience can simply guide suchcustomers to either to stop adversarial behavior or to stop visiting.

Additionally, the system and method may enable a monitoring system thatis more robust to adversarial attacks by reducing attack vectors.Someone trying to exploit a weakness of one monitoring system will bepresented with the challenge of simultaneously getting around asecondary monitoring system.

Similarly, the system and method may also provide a safe avenue for usercoaching when a user is unfamiliar with how to interact in anenvironment offering a user experience powered by the monitoringsystem(s). This may occur in CV-driven applications that enable checkoutfree shopping or expedited checkout where a customer accidentallyselects items in a way that results in poor accuracy. For example, anhonest user may unknowingly act in such a way that confuses a primarymonitoring system, but the system and method can facilitate identifyingsuch cases and presenting an opportunity for a worker or other agent toaid in resolving the issue. Such CV-driven applications are new forms ofhuman-computer interface and so multiple monitoring systems may serve toaid in coaching of usage. On the other hand, the system and method couldbe robust to repeated attempts to exploit a vulnerability.

As discussed, the system and method is preferably used in automation ofthe checkout process. Specifically, the system and method can be appliedto enabling automatic self-checkout where a customer is enabled toselect items for purchase and complete a checkout transaction withoutany or little interaction with a worker. The system and method can alsobe modified and applied in other fields of CV-driven applications. Inplace of evaluating items selected by users, the two or more monitoringsystems can be used in monitoring other events and activities. Examplesof other CV-driven applications include: item rental; securitymonitoring; interaction monitoring of a user for various areas ofoperations like commerce, retail, service providers, manufacturing,construction, healthcare, etc.; facial or biometric detection; roboticautomation/autonomous vehicles; and/or other fields of application.

2. Exemplary Implementations

There may be various embodiments of the system and method wheredifferent combinations of monitoring systems are used. Three exemplaryembodiments can include a primary CV monitoring system used with avariety of possible secondary monitoring systems, a point-of-sale (POS)system used along with a secondary CV monitoring system, and a userapplication for self checkout used along with a secondary CV monitoringsystem.

In the first exemplary embodiment, a secondary monitoring system is usedto validate the performance of a CV-based system using environmentalsurveillance. A variety of secondary monitoring systems may be used. Inone example, dispersed active inventory sensors such as RFID tags orsmart shelves with scales or targeted inventory sensors can be used toprovide higher accuracy view of a subset of items in the store. Inanother example, a user profile system can be used to provide a generalprediction of confidence level of the customer. In another, consumeroperated cameras (e.g., smart glasses) can form a first CV monitoringsystem and an environment installed imaging system can form a secondindependent monitoring system.

In the second exemplary embodiment of above, the CV-system can be usedas a secondary monitoring system to that of a POS system for worker orcustomer facilitated checkout. The CV-system in one variation can beused in detecting attempted theft and alerting a worker at the POSsystem of such activity. In another variation, this can be used as asetup for training and improving a CV-system and/or for on-boarding newproducts.

In the third exemplary embodiment of above, a user application can beused by a customer to manually enter items for purchase and used as aprimary monitoring system. In this implementation, the primarymonitoring system may be any suitable type of user controlledapplication such as a smart phone application, a smart watchapplication, smart glasses application, or any suitable type ofapplication for a computing device. In a related implementation, a smartcart or a similar hand held system could be used by a customer toautomatically, semi-automatically, and/or manually enter items forpurchase and used as a primary monitoring system. The secondarymonitoring system can be used in the background to validate if thecustomer is honestly, accurately, and correctly entering their selecteditems for purchase. A CV-system is preferably used as secondarymonitoring system. Alternatively, other secondary monitoring systemssuch as a partially dispersed RFID tags or smart shelves can be used inproviding some random sampling of activity and/or tracking high valueitems.

Such systems depend on cooperative users that can be trusted toaccurately, correctly, and honestly enter items. Such systems may havenumerous issues when used without use of the system and method such aswhen: a customer accidentally miscounting the number of items entered;customer thinking they entered an item when there was a system error; ora customer deliberately not scanning an item to avoid paying. Asecondary monitoring system could be used in a variety of ways toimprove these systems.

In one variation, portions of the primary monitoring system (e.g., asmart cart system) could forgo convoluted processes for majorvulnerabilities. For example, a CV-monitoring system could be used increating a record of items entered or not entered into the system.

In another variation, a secondary monitoring system with only partialcoverage of the environment may be used to provide redundant monitoringfor a subset of products thereby spot checking a customer's use of theprimary monitoring system or providing more robust monitoring for keyareas (e.g., for expensive/valuable products, difficult to monitorproducts, products with high shrinkage, etc.).

3. System

As shown in FIG. 1, the system for multiple-factor verification ofvision-based interactions preferably includes at least two monitoringsystems used in combination and a verification system 130 integratedwith the at least two monitoring systems. Each monitoring systempreferably generates a system output. The at least two system outputscan then be used in combination as processed by the verification system130 to establish monitoring alignment and/or a resulting interactionevent. The system may function to dynamically direct a variety ofsystems.

When applied to automating the checkout process, the two monitoringsystems can each be configured to generate a monitoring system outputrelated to selected items and/or user activity associated with itemselection. In some preferred monitoring systems, the system output is adetermined list of selected items (e.g., a virtual cart of items). Inother monitoring systems, the monitoring system output could be a listof possible or candidate selected items (but not necessarily selected),a list of a subset of selected items (e.g., only used in identifyingsome portion of the selected items), a user activity score (e.g., ascore related to “normal” shopping behavior), a user history score(e.g., historical evaluation of past activity), and/or other monitoringsystem outputs. Preferably, the monitoring systems generate a monitoringsystem output that is associated with a particular user. In other words,each customer in a store may have different interactions or eventsdetected and tracked by the at least two monitoring systems. As anexample of an alternative variation, the secondary monitoring systemcould track different events based on location and time withoutassociating to a particular customer. In this example, the time andlocation of interactions by a customer as indicated by a primarymonitoring system can still be evaluated for alignment by mapping therecords of the secondary monitoring system to corresponding records ofthe primary monitoring system.

At one or more instances, the system outputs from the two or moremonitoring systems can be used in combination and processed by theverification system 130 in evaluating monitoring alignment. Monitoringalignment can relate to: scoring the degree of agreement, identifyingdiscrepancies, predicting errors, generating an overall or product-levelconfidence level, generating a resulting virtual cart, evaluating someconditional logic based on properties of both the monitoring systems,and/or form any suitable assessment. The conditional logic coulddetermine an action corresponding to various properties or aspects ofthe two or more evaluations.

The monitoring alignment can be a metric reflecting overall accuracy ofthe monitoring or more specifically the cart modeling for a checkoutprocess. The monitoring alignment may alternatively include detailedevaluation of the accuracy of a checkout process. For example,individual items may be identified as being part of the cart or adiscrepancy between the monitoring systems. In some cases, the output ofthe verification system 130 may be alerts to special conditions or usedin triggering some interaction response. Monitoring alignment mayadditionally or alternatively be an updated modeling output. Forexample, an updated checkout list can be a final checkout list for usein a checkout process. The updated checkout list is preferably a moreaccurate list of items for purchase.

The system can be comprised of any two suitable monitoring systems.Preferably, one of the two monitoring systems is used as a primary andthe other monitoring systems are used as a secondary factor ofverification. The primary/secondary role of a monitoring system canadditionally change within the system. For example some regions of thestore may primarily rely on a first monitoring system 110 and use asecond monitoring system 120 as a secondary monitoring system, while ina second region, the first monitoring system 110 is the secondary systemand the second monitoring system 120 is used as the primary system.Alternatively, there may be no priority applied to the monitoringsystems, where the monitoring systems are used in a substantially equalcooperative manner.

A variety of combinations of monitoring systems can be used. Somepreferred monitoring systems that may be used in various combinationsinclude a CV monitoring system 141, an RFID-based monitoring system 142,a Smart infrastructure monitoring system 143, a user application 144,smart cart system, a POS monitoring system 145, a profiling predictionsystem, a security tag barrier system, and/or other suitable forms ofmonitoring systems as shown in FIGS. 2 and 2B.

CV Monitoring System

A CV monitoring system 141 functions to process and generate conclusionsfrom one or more sources of image data. The CV system can be configuredto perform person detection; person identification; person tracking;object detection; object classification (e.g., product identification);object tracking; extraction of information from device interfacesources; gesture, event, or interaction detection; scene description;and/or any suitable form of information collection using computer visionand optionally other processing techniques. More preferably, the CVmonitoring system 141 is configured to perform a sequence of the aboveprocesses in a coordinated manner for more advanced tracking. The CVmonitoring system 141 is preferably used to drive CV-based applicationsof an interaction platform. In the case of CV-driven retail, theCV-driven imaging system may facilitate generation of a virtual cartduring shopping, tracking inventory state, tracking user interactionswith objects, controlling devices in coordination with CV-derivedobservations, and/or other interactions. The CV-driven imaging systemwill preferably include various computing elements used in processingimage data collected by an imaging system. In particular, the CV-drivenimaging system is configured for detection of agents (e.g., people) andgeneration of a virtual cart based on interactions between people andproducts.

A CV monitoring system 141 will include at least one form of an imagingsystem that functions to collect image data within the environment. Theimaging system preferably includes a set of image capture devices. Theimaging system might collect some combination of visual, infrared,depth-based, lidar, radar, sonar, and/or other types of image data. Theimaging system is preferably positioned at a range of distinct vantagepoints and preferably forms substantially ubiquitous monitoring withinthe environment as described below. However, in one variation, theimaging system may include only a single image capture device. The imagedata is preferably video but can alternatively be a set of periodicstatic images. In one implementation, the imaging system may collectimage data from existing surveillance or video systems. The imagecapture devices may be permanently situated in fixed locations.Alternatively, some or all may be moved, panned, zoomed, or carriedthroughout the facility in order to acquire more varied perspectiveviews. In one variation, a subset of imaging devices can be mobilecameras (e.g., wearable cameras or cameras of personal computingdevices).

In one variation, a subset of imaging devices can be mobile cameras(e.g., wearable cameras or cameras of personal computing devices). Forexample, in one implementation, the CV monitoring system 141 couldoperate partially or entirely using personal imaging devices worn byagents in the environment. The image data collected by the agents andpotentially other imaging devices in the environment can be used forcollecting various interaction data.

In a preferred implementation, at least a subset of the image capturedevices are oriented for over-head monitoring, wherein the image capturedevices collect a substantially aerial perspective. In a shoppingenvironment, the imaging system preferably includes a set of staticallypositioned image devices mounted with an aerial view from the ceiling.The aerial view imaging devices preferably provide image data acrossstored products monitored for virtual cart functionality. The imagesystem is preferably installed such that the image data covers the areaof interest within the environment (e.g., product shelves). In onevariation, imaging devices may be specifically setup for monitoringparticular items or item display areas from a particular perspective.

Herein, ubiquitous monitoring (or more specifically ubiquitous videomonitoring) characterizes pervasive sensor monitoring across regions ofinterest in an environment. Ubiquitous monitoring will generally have alarge coverage area that is preferably substantially continuous thoughdiscontinuities of a region may be supported. Additionally, monitoringmay monitor with a substantially uniform data resolution. In somevariations, a CV monitoring system 141 may specifically have an imagingsystem with only partial coverage within the environment. For example, aCV monitoring system 141 used for secondary monitoring may generate apartial checkout list.

Large coverage, in one example, can be characterized as having greaterthan 95% of surface area of interest monitored. In a shoppingenvironment, this can mean the shelves and product displays as well asthe shopping floor are monitored. Substantial uniform data resolutionpreferably describes a sensing configuration where the variability ofimage resolution and/or coverage of different areas in the environmentare within a target range. In the exemplary case of automatic checkoutCV-driven applications, the target range for image resolution issufficient to resolve product-packaging details for productidentification.

Ubiquitous monitoring may optionally include the characteristic ofredundant monitoring. This may involve having redundant coverage frommultiple vantage points. For example, an item on a shelf may be visibleby two different cameras with adequate product identification resolutionand where the cameras view the item from different perspectives. In anenvironment like a grocery store this could mean 10-200 camerasdistributed per an aisle in some exemplary implementations. Ubiquitousmonitoring may be particularly used in scenarios where the CV monitoringsystem 141 is used as a primary monitoring system. Though ubiquitousmonitoring is not a requirement for all variations described herein.

Similarly, the system may additionally include other computer input oroutput devices across an environment. The system and method can be usedin the collection of sensor data and/or generation of an output inaddition to or as an alternative to video and/or image data. Other formsof devices such as microphones, Bluetooth beacons, speakers, projectors,and other suitable devices could additionally or alternatively beintegrated into system modules that may be installed across anenvironment. These additional sensors in some variations may be used incombination within a single monitoring system.

In a store environment (e.g., a grocery store), the CV monitoring system141 can be configured to additionally track a checkout list forautomatic checkout and/or expedited checkout at a checkout station. Inone variation, the CV monitoring system 141 may be used to generate avirtual cart, which may be performed in a manner substantially similarto the system and method described in US Patent Application publicationNo. 2017/0323376, filed 9 May 2017, which is hereby incorporated in itsentirety by this reference. In other settings like an industrial,office, or hospital setting, the CV monitoring system 141 may be used tomonitor worker actions and operations. In environments like a gym orother areas it may track activity. Herein, the use case of tracking itemselection for facilitating checkout is used as a primary example, butthe system is not limited to such uses.

The CV monitoring system 141 can include a CV-based processing engineand data management infrastructure. The CV-based processing engine anddata management infrastructure preferably manages the collected imagedata and facilitates processing of the image data to establish variousmodeling and conclusions relating to monitoring objectives. For example,the selection of an item and the returning of an item are of particularinterest. The data processing engine preferably includes a number ofgeneral processor units (CPUs), graphical processing units (GPUs),microprocessors, custom processors, FPGA's, and/or other computingcomponents. The computing components of the processing engine can residelocal to the imaging system 112 and the environment. The computingresources of the data processing engine may alternatively operateremotely in part or whole.

The CV monitoring system 141 may additionally or alternatively includehuman-in-the-loop (HL) monitoring which functions to use humaninterpretation and processing of at least a portion of collected sensordata. Preferably, HL monitoring uses one or more workers to facilitatereview and processing of collected image data. The image data could bepartially processed and selectively presented to human processors forefficient processing and tracking/generation of a virtual cart for usersin the environment.

In a preferred embodiment, the CV monitoring system 141 is a primarymonitoring system that is validated by a secondary system. The secondarysystem can be used in combination with the CV monitoring system 141 tospot check the virtual cart from the CV monitoring system 141, to assessthe likelihood of the virtual cart, or make any suitable type ofdetermination.

The CV monitoring system 141 may alternatively be used as a secondarysystem. As a secondary system, the requirements and constraints may beless for the CV monitoring system 141. Accordingly, the CV monitoringsystem 141 could provide a range of general visually extractedassessments of person activity. As opposed to providing a fullorchestration of CV monitoring that interprets multiple complex tasks,the CV monitoring system 141 could provide base CV-driven modeling suchas event tracking, people-object proximity tracking, and the like.Similarly, a secondary CV monitoring system 141 may not have fullcoverage within the environment and include an imaging system withpartial coverage of the environment. As such the CV monitoring system141 may not include large-scale imaging device coverage. In such ascenario the output of the CV monitoring system 141 (e.g., the systemevaluation) may be a partial product selection list.

In one variation, the CV monitoring system 141 may be used as asecondary monitoring system and configured to generate a virtual cart asabove but with lower accuracy requirements. This can be used tospot-check a primary monitoring system such as a POS system, a userapplication, and/or an alternative implementation of a CV monitoringsystem 141.

In another variation, the CV monitoring system 141 may be used as asecondary monitoring system and configured to create a list of candidateproducts. The candidate products may provide some conditions forproducts identified in a primary monitoring system.

In one variation, the CV monitoring system 141 may be used as asecondary monitoring system to generate a shopping behavior evaluationfor a user. In one variation, generating a shopper behavior evaluationcan include tracking customer location within an environment, which maybe used to deduce particular patterns. For example, the motion paththrough the store of a user may indicate some probability of buying somenumber of items or particular types of items. In another variation,generating a shopper behavior evaluation can include detecting productinteractions. For example, a generalized list of probably productinteraction events may be tracked per customer in the store. Productinteraction events may be a record of any suitable type of productinteraction. In one variation, the list of product interaction eventsmay not positively identify when a product was picked up and kept by thecustomer. However, a list of possible product interactions may be usedalong with a primary monitoring system that definitively tracks itemselection to draw some conclusions of customer behavior. In anothervariation, generating a shopper behavior evaluation can includeconfiguration to detect a set of suspicious interaction events.

Additionally, a CV monitoring system 141 may transition between statusas a secondary and primary monitoring system. This can be used foronboarding a system in a new store. Primary/secondary status mayadditionally be dynamically invoked based on different conditions. Forexample, if the CV monitoring system 141 enters a particular monitoringstate for a particular customer, the CV monitoring system 141 may bechanged to a secondary monitoring system and a new monitoring system maybe used as the primary monitoring system.

RFID-Based Monitoring System

An RFID-based monitoring system 142 functions to use tracking of tagsattached to items for tracking item position/activity. More generally,the system may include a radio frequency tag monitoring system. A radiofrequency (RF) tag monitoring system may not include uniquelyidentifiable tags. As one example, a security tag barrier system may bea RF tag monitoring system where there is only one “identity” associatedwith the tags used in the environment. Alternatively, the security tagbarrier system can be an RFID-based monitoring system 142. Herein,RFID-based monitoring system 142 s are primarily used as the example,but one skilled in the art could recognize that alternative systems suchas a general RF tag monitoring system, a Bluetooth beaconing system,some active beaconing, acoustic beaconing, or other suitable forms ofphysical tag monitoring may be used.

An RFID-based monitoring system 142 preferably includes at least oneRFID reader and a set of RFID tags detectable by the RFID reader. TheRFID tags may be passive or active. The RFID tags are preferablyattached to items in the environment such that the coupled items can betracked through the environment. The RFID readers could be placed in anysuitable location to track location. RFID readers could be integratedwith product storage/shelving, checkout regions, exits and entrances,and/or at any suitable location. Preferably, a distributed network oftag readers is used, which in some variations may be used to triangulatetag position.

In one implementation, the RFID tag system is used in selectivemonitoring of a subset of the inventory items. For example, a samplingof different inventory items may be tracked through the RFID system. Theitems that are tracked though provide a sanity check to the primarysystem. The RFID-based monitoring system 142 could alternatively be aprimary monitoring system.

An RFID-based monitoring system 142 is preferably used as a secondarysystem. The RFID-based monitoring system 142 could involve a set ofproducts adding a tag, detecting a set of tags (e.g., one or more) in aparticular region. For example, if a subset of the products selected bythe customer include tagged products detectable by the tag readingsystem, then an at least partial product selection list could begenerated.

Smart Infrastructure Monitoring System

A Smart infrastructure monitoring system 143 functions to perform otherforms of product, equipment, or people tracking. Smart infrastructuremonitoring system 143 may include or be used alongside an RFID-basedmonitoring system 142. A Smart infrastructure monitoring system 143 caninclude digital scales, proximity sensors, mechanical sensors, or othersuitable forms of sensing. In one particular variation, digital scalesand/or item tracking sensors can be integrated with shelving units tomonitor individual items during storage. The Smart infrastructuremonitoring system 143 is preferably configured to track item removalfrom a shelving unit. The Smart infrastructure monitoring system 143 mayalso track that an item was removed by an individual person or by apossible set of people. In one variation, a removed item is positivelyassociated with a detect user. In this way, the Smart infrastructuremonitoring system 143 may generate an at least partial checkout list ofa customer. In another variation, a removed item may be simply betracked as having been selected within the store (possibly within aparticular time window). This may be used to cross reference itemselection as detected by a different monitoring system.

A Smart infrastructure monitoring system 143 may be used as a secondarymonitoring system as with the RFID system. In some implementations, theSmart infrastructure monitoring system 143 can be used in combinationwith the RFID-based monitoring system 142 in providing selectivemonitoring of a subset of items in an environment. The scales and sensorfusion approaches to monitoring may be used in select areas or forselect products such as products commonly purchased, with a high value,or selected for any suitable reason.

User Application Monitoring System

A User application monitoring system 144 functions to utilize userparticipation in generating a model of their activity. In the checkoutuse case, a user application preferably describes any checkout systemthat promotes user-facilitated adding of items to a checkout list. Inone particular implementation, a User application monitoring system 144is an app on a smart phone that allows a user to enter product items forpurchase using CV detection of a viewed item, scanning of a barcode orproduct identifier, and/or manually entering a product listing. The“virtual cart” of the shopper can be fully managed by the customer. Asthis relies on customer trust, a secondary monitoring system can providea form of verification.

A User application monitoring system 144 of alternative use cases mayfacilitate accounting for other forms of user activity. This may be auser application on a smart phone, or wearable computer, or othersuitable device.

In some embodiments, a User application monitoring system 144 can beused as a primary monitoring system that is validated through asecondary monitoring mode. In one preferred implementation, a userapplication may be used as a primary monitoring system, and a CVmonitoring system 141, an RFID-based monitoring system 142, a Smartinfrastructure monitoring system 143, and/or other form of monitoringsystem can be used in assessing the validity of what the user claims tobe purchasing as indicated by the app, and what has been detected by thesecondary monitoring system.

As a related variation, the system could include a smart cart monitoringsystem 147, wherein shopping baskets, carts, or other item holdingdevices may be used to track item selection as items are stored in thecorresponding vessel. A smart cart monitoring system 147 could be usedin a similar manner as User application monitoring system 144 s. A smartcart monitoring system 147 may be used to automatically,semi-automatically, and/or manually enter items for purchase and used asa primary monitoring system. A smart cart monitoring system 147 mayinclude a cart or vessel used to hold items and some item sensingsystem. The item sensing system may be a local CV monitoring system 141,RFID monitoring system, or other suitable automatic sensing systems. Thesmart cart may additionally include more basic sensing elements such asa barcode scanner, a digital scale, an internal measurement unit orother suitable motion/orientation sensing elements (e.g., accelerometer,gyroscope, altimeter, magnetometer, etc.), and/or other suitableelements. The smart cart preferably includes a communication module towirelessly communicate with the rest of the system. In yet anothervariation, the smart cart may include a user interface such that a userapplication may be operated on the cart.

While the smart cart may be used in many scenarios as a primarymonitoring system, the smart cart could alternatively be used in asecondary monitoring capacity. Use as a secondary monitoring system mayenable the smart cart monitoring system 147 to be configured for anarrower scope of monitoring. In one variation, wirelessly connectedscale at the bottom of a smart cart could be configured to wirelesslyrelay scale activity data (e.g., changes in weight) to a central system.

POS Monitoring System

A POS monitoring system 145 functions to provide data collected from asystem setup for point of sale transactions such as those used forworker facilitated checkout or self-checkout kiosks. Data collected froma POS monitoring system 145 can be used in a variety of ways with acomplementary monitoring system.

The POS monitoring system 145 could be an existing POS checkout stationsuch as one staffed by a worker or a self-checkout kiosk. A POS checkoutstation used with the system is preferably connected or includes acommunication interface such that items entered into the POS system canbe detected and communicated to the verification system 130 or anothersuitable remote computing system. In another variation, the POSmonitoring system 145 may be an “add-on” device or system that canextract POS-based information. In one implementation, a barcode scannerproxy module can be inserted in between barcode entry devices and thecentral processing unit of a POS system. The barcode scanner proxymodule can preferably establish a data connection to the verificationsystem 130.

In one variation, a POS monitoring system 145 can be a primary systemthat is used in driving the transactions. A secondary system, like asecondary CV-driven system, can be used to detect mistakes in operatingthe POS system or to detect bad actors. A POS monitoring system 145could similarly be used as a secondary system.

Profile Prediction System

A Profile prediction system 146 functions to use a predictive engine toprovide context of evaluating another monitoring system. The profilingsystem is preferably used as a secondary system. Preferably, a Profileprediction system 146 can use data of a particular person, class ofperson, the store, or other data scopes to generate predictions and/orassess likelihood of predictions from another monitoring system. Inparticular, the Profile prediction system 146 can use shopping historyto determine likelihood of accuracy of a checkout process as detected bya primary system. In one variation, the Profile prediction system 146outputs. For example, customers that historically attempt to mislead thesystem may be more likely to do so again. In another example, customersthat historically confuse the system due to legitimate shopping habitsmay also be more likely to do so again. In another example, a shopperselecting items that are predictively aligned with past behavior isprobably being monitored accurately, and a shopper modeled as havingselected items by a primary system may be determined as likely notmodeled accurately if those items are not aligned with expected purchasepatterns.

A profile can additionally be updated for different visits/shoppingexperiences. In some cases, the Profile prediction system 146 may evenbe used as a tertiary monitoring system.

The Profile prediction system 146 may use a variety of types of datasuch as purchase history, completeness of a profile (e.g., address,name, email, phone number, etc.), age verification, premium membership,social media account connection, biometric profile, payment information,and/or other details. In some implementations, users with more completeand verified information will be more accountable and therefore may beless likely to deliberately mislead the system.

These various monitoring systems may be used in any suitablecombination.

The verification system 130 functions to coordinate evaluation ofmonitoring alignment between the at least two monitoring systems. Asdiscussed above, the system is preferably implemented in connection withanother computer-based system such as an enhanced checkout system forretail environments, which may provide features like checkout-freeshopping, accelerated checkout processing at a POS station, theftprevention monitoring, and the like.

The system may include a data management system and other supplementarysystem elements of a computing platform, service, or application. Thesystem may additionally include or interface with a user applicationand/or connected computing devices.

The data management system functions to store user-records and/oruser-account information. The data management system may additionallymanage, include, or interface with the verification system 130.

User Applications and/or connected computing devices function as clientdevices that may participate within the interactions of the system. Auser application instance or a connected computing device are preferablyaddressable endpoints within the system such that the data managementsystem or an alternative component can transmit data or information canbe communicated to the user application instance or the connectedcomputing device. An SDK and/or API may be provided such that thirdparty applications can establish such integration with a CV monitoringsystem 141 and/or platform. In one variation, the system can include aworker application that connections to a service of the platform suchthat worker directions can be communicated to the worker.

4. Method

As shown in FIG. 3, a method of a preferred embodiment can includeexecuting a first monitoring system and thereby generating a firstevaluation S110, executing a second monitoring system and therebygenerating a second evaluation S120, determining monitoring alignmentS130, and triggering an action in response to the monitoring alignmentS140.

The method is primarily described as it could be applied to retail. Acustomer is generally a human shopper but could alternatively oradditionally include shopping carts, a shopping bag, an automatedmachine, and/or any suitable tracked entity. The method couldalternatively be applied to any sensor/prediction driven applicationthat may be retail related or outside of retail. The method ispreferably applied in the situation where one of the first monitoringsystem and the second monitoring system is a computer vision monitoringsystem. More preferably, the computer vision monitoring system is acomputer vision monitoring system with multiple cameras distributedacross the environment.

The method is preferably implemented by a system substantially similarto the one described above, but the method may alternatively beimplemented by any suitable system. Any suitable combination ofmonitoring systems can be used, and any suitable use as a primary orsecondary monitoring system can be used. Preferably, the firstevaluation is generated independent from the second evaluation, and thesecond evaluation is generated independent from the first evaluation. Insome variations, where more than two monitoring systems are used,multiple groups of monitoring systems may be used in combination, butpreferably at least one monitoring system is independent from a subsetof other monitoring systems.

Block S110, which includes executing a first monitoring system andthereby generating a first evaluation, functions to generate aprediction or collect some data metric using a first approach. Morespecifically, block S110 can function to collect a primary system outputassociated with customer selection of items. Herein, the use of firstand second is not intended to imply primary or secondary, but forconsistency, examples generally refer to the first monitoring system asthe primary monitoring system (if there is a primary system).

The first evaluation is preferably a data output from the firstmonitoring system. The evaluation of one preferred variation is anevaluation of selected items by a customer. The evaluation of selecteditems can include a list of items planned to be used in completing apurchase transaction. This may form the checkout list or “virtual cart”of an automatic checkout shopping experience. Different implementationsmay use different monitoring systems.

The first evaluation may alternatively be any suitable type of outputfrom the monitoring system, which depends on the type of monitoringsystem. The evaluation may alternatively be a classification of a user,a history of interactions by a user, a map of tracked position during avisit to an environment, and/or any suitable type of evaluation.

In one variation, a CV monitoring system is used in collecting imagedata, applying a CV-based processing, and generating the firstevaluation. CV-based processing functions to model item and customerinteractions based on the image data. Preferably a variety of techniquesmay be used. Preferably, applying CV-based processing can includetracking a set of users through the environment; for each user,detecting item interaction events, updating items in a checkout listbased on the item interaction event (e.g., adding or removing items).The checkout list can be a predictive model of the items selected by acustomer, and, in addition to the identity of the items, the checkoutlist may include a confidence level for the checkout list and/orindividual items. The checkout list is preferably a data model ofpredicted or sensed interactions. Other variations of the method mayhave the checkout list be tracking of the number of items possessed by acustomer or detection of only particular item types (e.g., controlledgoods like alcohol, or automatic-checkout eligible goods). The CVmonitoring system may use algorithmic approaches applying traditionalcomputer vision techniques, deep learning models, machine learning,heuristic modeling, and/or other suitable techniques in processing theimage data. The CV monitoring system may additionally use HL inevaluating image data in part or whole.

In another variation, an RFID-based monitoring system may be used indetecting the selection of an item and assigning the selected item to acustomer through RFID tag monitoring. More generally an RF monitoringsystem includes detecting tag presence and associating tag presence toproduct presence in one or more regions. This may further be extended totracking a tag through a number of locations or throughout anenvironment (e.g., using triangulation of an active RFID tag). Forexample, a product may include an attached or embedded RFID tag that canbe detected as being moved into proximity of the customer. In somecases, the customer may be operating a device with an RFID reader orother suitable RF-enabled device such as a smart cart. Smartinfrastructure monitoring can additionally or alternatively be used.More generally, an RFID-based monitoring system may include detectingpresence of a tag associated with an item. This can be associated withthe presence of an item outside of its normal location (e.g., theshelf), presence in a particular zone (e.g., checkout region, storeexit), possession by a user, or any suitable type of conclusion.

As with some other monitoring system variations, an RFID-basedmonitoring may not have a direct way of assigning detected items to aspecific user or tracking a user through the store. In oneimplementation, the RFID-based monitoring system interrogates a user fordetectable RFID tags at the point of interaction such as at a checkoutstation, wherein there is no explicit mapping to a user and it is morebuilt into the interaction that the associated user will be present.

In another variation, a user application monitoring system may depend onuser interaction in part to establishing a user indicated selection ofitems. A user application monitoring system variation may involvecollecting item selection through a user interface and generating acheckout list. For example, a user may scan barcodes of products forpurchase, manually enter product information, select products from amenu, or enter items for purchase in any suitable manner. A smart cartcould similarly generate a checkout list. A smart cart may use a varietyof sensor to identify products.

In another variation, a POS monitoring system such as one that can beused in a checkout aisle of a store may be used to provide the firstevaluation. When a worker operates the POS monitoring system, the firstevaluation can be a worker-based evaluation of selected items.

Block S120, which includes executing a second monitoring system andthereby generating a second evaluation, functions to develop a paralleland/or complementary evaluation. The first evaluation is preferably usedin validating the first monitoring system and/or flagging particularscenarios warranting concern. In some cases, the secondary monitoringsystem may be a partial monitoring system that does not provide fullmonitoring of an environment. In other words, the secondary monitoringsystem may provide spot-checks. The primary and secondary monitoringsystems may alternatively not be assigned a priority wherein the notionof primary and secondary does not impact weighing or prioritization ofone over the other. Preferably, the attack vectors of the secondarymonitoring system are not aligned with those of the primary monitoringsystem.

The type of evaluation can be the same, similar, or alternativelydifferent. Evaluation type characterizes the format and content of anevaluation. When the type is shared, the first and second evaluationsmay have similar data such as when creating a virtual cart of predicteditems selected by a user. Accordingly, in one variation, the secondevaluation is an evaluation of customer selection of items similar tothe first evaluation. The type may also be of similar types. Forexample, a first evaluation may attempt to be a full representation ofselected items, and a second evaluation may attempt only a partialrepresentation of selected items. The evaluations can also be ofdifferent evaluation types.

In one variation, a first evaluation may attempt to be a fullrepresentation of selected items, and a second evaluation may representa sub-property of interaction activity. The full set of properties of aninteraction activity is preferably sufficient to generate somecharacterization of the interaction. For a monitoring system that cangenerate a reasonable approximation of a full checkout list wouldcapture the full set of properties of shopping interaction. Asub-property of interaction activity characterizes one or a subset offactors that can be used in describing the full interaction. Examples ofa sub-property that may be detected by a second monitoring system caninclude user location, occurrence of events without specific labeling,and the like.

In another variation, a second monitoring system may characterizequalitative assessments such as the level of caution to treat a customerbased on detected behavior and/or past profile history.

The second evaluation, especially when used in a non-primary capacity,may be a data output with partial reporting of the events of interest.For example, the second evaluation may make no attempt to fully predicta checkout list for automatic checkout. In one variation, the secondevaluation may provide a partial list of items detected likely in thecheckout list. This may be generated by systems like the RFID-basedmonitoring system or smart infrastructure monitoring systems where thesystems perform partial monitoring of the products in an environment. Inanother variation, the second evaluation may provide a probabilityrating of a wide range of products. This may be generated by aCV-monitoring system or an alternative monitoring system to trackcustomer location through the store such that product likely-hood can bemodeled based on the regions visited by the user. In another variation,a profile prediction system may be used as a generalized check on theprimary monitoring system.

Block S130, which includes determining monitoring alignment, functionsto use the first and second evaluations to make a determination.Determining monitoring alignment is preferably performed in associationwith a particular customer or alternatively checkout instance (e.g., thecheckout process of a group of users). The first and second evaluationsare analyzed in combination.

In one variation where the first and second evaluations both haveevaluations of the same or similar type, block S130 may includecomparing item selection evaluations and detecting discrepancies. For agiven customer, determining monitoring alignment can include comparingselected items associated with a first customer in the first evaluationto selected items associated with the first customer in the secondevaluation. In this variation, determining monitoring alignmentfunctions to compare across all items of a checkout list. This may beused to detect errors or issues where the two item selection evaluationsdo not fully align. For example, if the first and second monitoringsystems predict different items there is a cause for a concern. Fullalignment across a list of selected items would be an ideal scenario andmay be used to allow or enable interactions with the computing systemsuch as automatic checkout (e.g., the automatic execution of atransaction for the modeled selected items). Determining monitoringalignment can additionally resolve discrepancies such that a perfectmatch is not always a requirement for interactions such as automaticcheckout. In one variation, an evaluation may characterize confidencelevels for predictions of individual items or the list in general may beprocessed between the two. Such confidence levels could be incorporatedinto determining the monitoring alignment. Additionally, the itemproperties of mismatched items in the first and second evaluations maybe incorporated into determining the monitoring alignment. For example,the modeling of a jar of pickles with low confidence in one evaluationmay be resolved with the modeling of a jar of relish with highconfidence. Product visual similarity and high confidence in one maystill indicate high alignment between the two monitoring systems.

In some variations, the second monitoring system may only provide apartial monitoring of relevant interactions. As such a one-to-onecomparison of items would not be performed. Determining monitoringalignment would preferably include verifying that a set of selecteditems associated with a first customer of the second evaluation maps toat least a subset of selected items associated with the first customerof the first evaluation. In an implementation with a CV-monitoringsystem and an RFID-based and/or smart infrastructure monitoring system,the CV monitoring system may be configured to target providing a fullrepresentation of items selected by a customer. The secondary monitoringsystem may be used to spot-check these items. In some cases, theenvironment may be configured so key items (e.g., expensive items oritems commonly stolen) can be monitored by the secondary monitoringsystem. For example, a store may select to add RFID tags to a subset ofproducts. During a checkout evaluation for a customer, three itemsdetected of an RFID-based monitoring system are preferably mapped tocorresponding items of an evaluation from a CV monitoring system.

In another variation, a qualitative evaluation relating to confidence,trust, or levels of caution may be used to determine how to assessanother evaluation. For example, a virtual cart generated by a CVmonitoring system will generally have confidence levels associated withthe various items predicted as having been selected. The confidencelevel thresholds may be augmented in response to an evaluation from aprofile predictive system.

In an implementation with a primary user application monitoring systemand a secondary CV monitoring system, the CV-system can operate toproduce a redundant virtual cart for a customer. The redundant virtualcart can be compared to the virtual cart managed by the user and used toprovide some security around user trust.

Various other implementations can similarly use the at least twoevaluations in combination.

Block S140, which includes triggering an action in response to themonitoring alignment, functions to use the monitoring alignment to takesome action. Some responses may be active in that they can directlyalter the checkout process and/or system actions during a shoppingexperience. Other types of responses may be passive in that they triggera background action, which may or may not impact subsequent or previousinteractions.

Some examples of response actions can include enabling or disabling acheckout process, directing customer handling, scoring a customerprofile, providing an operation instruction, updating at least one ofthe monitoring systems, controlling a device, and/or other suitableactions. Multiple actions could additionally be triggered at differenttimes.

In one variation, enabling or disabling checkout process functions toregulate the checkout process. More generally enabling or disabling thecheckout process is part of controlling a checkout process incoordination with the monitoring alignment. Controlling a checkoutprocess preferably involves permitting a checkout process at a checkoutsystem for items of at least one of the first or second evaluation whenmonitoring alignment satisfies an alignment condition. Conversely,controlling the checkout process may involve executing a regulatingaction to the checkout process which may halt or alter a checkoutprocess, signal some issue to a worker or device, or perform anysuitable action to address a potential issue. As one preferredimplementation, controlling the checkout process will includecommunicating an alert to the checkout system when monitoring alignmentdoes not satisfy the condition.

Control of a checkout process is preferably applied as a customer isending a shopping session (e.g., leaving a store, entering a checkoutzone, etc.) The checkout process may be selectively disabled or enabledbased on the result. Specifically, regulating a checkout processpreferably includes communicating to a checkout system the status of acheckout attempt. For example, when integrating with a POS system, anerror message may be communicated to the POS system preventing orhalting the checkout process. For example, a customer trying to useautomatic self checkout when paying at a payment kiosk may be preventedfrom completing the checkout process at that station and may be directedto speak to a customer representative. Enabling or disabling a checkoutprocess may in turn determine directing of a customer, messaging to acustomer, controlling some security system, or other systems.

In another variation, directing customer handling can function to alterthe operational interaction with a customer. In some cases, this mayinvolve influencing how a human worker response to a customer. Thisvariation may include generating worker instructions based on themonitoring alignment and communicating the worker instructions to aworker management device that guides a worker in inspecting customers.The worker instructions are preferably generated and/or communicatedwhen monitoring alignment does not satisfy an alignment condition. Theworker instructions may be directed at a variety of tasks. Somepreferred uses could involve directing how and when to inspect acustomer. There may be various forms of inspection. For an improved userexperience the system may be biased towards specifying the leastintrusive inspection option appropriate to address the alignment issue.For example, the worker instructions may: direct inspection of one itemor a subset of items that have monitoring misalignment; direct fullcheckout list verification; direct performance of a cursory checkout(e.g., low intrusion but signaling vigilance), directing the customer toa particular checkout station to repeat/perform checkout, or anysuitable action.

The worker management device can be an application or device used by aworker to assist in monitoring the environment. In one implementation,the application can be executable on a phone or wearable computer (e.g.,connected audio device, smart watch, smart glasses, etc.) where alertsor notifications can be triggered when a customer, bag, or item shouldbe inspected. In another implementation, the worker management devicecould be a dedicated device that signals events or conditions. Forexample, a worker management device could be a basic connected devicethat lights up one or more different light signals to indicateconditions like “permit customer”, “spot check customer”, and “stopcustomer”.

Worker instructions can proactively promote a distribution ofinspections across customers. The determination to inspect a customermay be customized for different customers. Customer handling can bebased entirely on comparison of the first and second monitoring systems,but may additionally integrate balancing checks across the population ofcustomers.

During an inspection, the method may include collecting a thirdevaluation of item selection, which is preferably used in modifying thelist of selected items so that a purchase transaction can be completed.The third evaluation can additionally be used as training feedback tothe first and/or second monitoring systems.

In another variation, scoring a customer profile can function to updatea record associated with a customer. In some cases the history ofmonitoring alignment may be used determine handling in subsequentvisits. When monitoring alignment is outside of the norm, a user'shistory of monitoring alignment can be used as a signal to morecautiously evaluate subsequent checkout processes for that user. Scoringa customer profile preferably involves updating or creating a userprofile of a customer. The user profile is preferably updated withinformation about the monitoring alignment. In one variation, a historyof monitoring alignment or particular issues with an evaluation can berecorded in the user profile. Repeated occurrences of monitoringalignment may trigger actions in subsequence visit. Accordingly, themethod may additionally include a customer during a subsequent visit,accessing a stored user profile of the customer, and triggering anaction based in part on the user profile. This may include augmentingcheckout processing directing worker actions or any suitable action.

In another variation, providing an operation instruction can function togenerate a recommendation or instruction used in altering how theenvironment is run. The operation instructions or reports may begenerated and used in augmenting the store or products to enhanceperformance of the one or both monitoring systems. The operationinstruction may relate to a product(s) or the store in general.Operation instructions may include an operational instruction thatspecifies at least one of: item stocking placement (or other stockingvariables), configuration of at least one of the first or secondmonitoring systems, which items to monitor, pricing of items, and/orother forms of instructions. As one example, an operation instructionmay include directing packaging of products or marking of products.Select products may be identified as benefiting from visual packagingchanges. In some cases, the form of visual augmentation may also berecommended. For example, placing stickers on one type of product mayimprove detection. This may be used to enhance the performance of a CVmonitoring system. As another example, an operation instruction mayinclude directing positioning of products. Rearranging the stocking ofproducts may result in better performance when close proximity leads toincreased errors. RFID tagging or placing for monitoring by smartinfrastructure may other forms of recommendations.

In yet another variation, updating at least one of the monitoringsystems functions to use the result of the monitoring alignment asfeedback into one or both monitoring systems. This can be particularlyuseful in training systems such as CV monitoring system using anothermonitoring system like a POS monitoring system. For example, triggeringan action can include updating the computer vision monitoring system inresponse to the monitoring alignment. The training model, configurationof imaging devices (e.g., position, picture/video quality). In onevariation, results of a checkout process can be used to train themonitoring systems. In another variation, collected data resulting in amonitoring error can be flagged for inspection or retraining.

As described herein, there are multiple permutations of monitoringsystem combinations, which may be applied for a variety of objectivesand triggering different types of tasks. Some permutations are reflectedin the exemplary implementations of FIGS. 4-8 and those described below,but the method is not limited to these implementations.

One exemplary variation can use a CV monitoring system as a firstmonitoring system and a RF monitoring system as the second monitoringsystem. In one variation, this may be applied to spot check a checkoutlist of the CV monitoring system using a partial item list of the RFmonitoring system. Accordingly, the RF tag monitoring system can be anRFID tag monitoring system with an identifier mapped to at least oneproduct. In this variation, the second evaluation can be an at leastpartial item selection list, and determining monitoring alignment caninclude comparing the at least partial item selection list to a selecteditem list of the first monitoring system. As shown in FIG. 5, analternative variation may be applied to directing customer handlingwhich is generally manifested through delivery of worker instructions.

In another variation, a CV monitoring system can be used in combinationwith a security tag detector barrier, which is used for monitoringtheft/shrinkage detection. The security tag detector barrier ispreferably a device positioned at an exit/entrance of the environment orany suitable region where it may be beneficial to checking for itemremoval. This variation may function to provide an enhanced tagdetection barrier system that more intelligently generates an alert. TheCV monitoring system can provide additional context to prevent falsealarms (e.g., disabling an alarm when a product was purchased orproperly tracked for automatic checkout) and/or to expand productmonitoring beyond just items that have an attached tag. In oneimplementation, the security tag detector barrier generates anevaluation that is a binary signal that corresponds to tag presencewithin a monitored region of the barrier. In some implementations, itemidentity, proximity to the barrier and/or other data may be detected andused in coordinating when actions are triggered. Preferably triggeringan action can include triggering an alarm of the security tag detectorbarrier based on monitoring alignment as shown in FIG. 6. This alarm maybe used to signal when an item is being removed without payment.Alternatively, a positive alarm (more likely a “signal”) can indicatewhen a customer is verified and approved for exiting. For example, as acustomer approaches an exit the barrier may signal approval with alight, actuating a physical barrier (e.g., opening a door or lifting agate).

In this variation, custom logic may be applied when evaluatingmonitoring alignment. For example, if a tag is detected and theevaluation of the CV monitoring system does not have a record matched tothe tag an alarm may be triggered. If a tag is detected and theevaluation of the CV monitoring system has a high confidence rating,then an alarm may not be triggered. Similarly, if a tag is detected andthe evaluation of the CV monitoring system has accounted for the system,then the alarm may not be triggered (and may be actively disabled toprevent an alarm triggered by the tag). If the CV monitoring system haslow confidence, an alarm may be triggered based on user proximity to thebarrier regardless of tag presence. If the evaluation of the CVmonitoring system indicates detection or likely possession of an itemfor which payment has not or cannot be processed, an alarm may besignaled. This CV triggered alarm can allow monitoring of products thataren't tagged or may not be able to be tagged (e.g., small items).

In another variation, a smart cart monitoring system can be used incombination with a CV monitoring system, which is used for at leastpartially redundant checkout list comparisons. In one variation, eachmonitoring system may attempt to provide a valid checkout list for anentire shopping experience of a user, wherein the monitoring alignmentis a comparison of ideally redundant checkout lists. This checkout listcomparison can be used in controlling a checkout process as shown inFIG. 4. Differences between checkout lists can indicate issues. Inanother variation, one monitoring system may generate a partial itemselection list and captures at least a partial set of the checkout listpredicted by the other monitoring system. In one implementation, the CVmonitoring system is used to spot-check the smart cart system, whereindetermining monitoring alignment partially verifies the item-based cartcontents against partial item selection list from the CV monitoringsystem.

In another variation, a user application can be used in combination witha CV monitoring system, which can be used for at least partiallyredundant checkout list comparisons or other forms of checkout listverification. Executing the user application can include collecting itemselection through a user interface and generating a checkout list. Aswith the smart cart variation above, the user application may bevulnerable to various types of attacks or unintentional error during useof the application. A computer vision monitoring system with partialimaging coverage of the environment (i.e., not every product isspecifically monitored). Executing the first monitoring system comprisesgenerating a partial item selection list; and wherein determiningmonitoring alignment partially verifies the checkout list contentsagainst the partial item selection list. In another variation shown inFIG. 7, CV monitoring can be used to track a subproperty of themonitored interaction (or other suitable type of monitoring output) thatrelates to selecting items for a checkout process. For example, a mappedpath through the environment of a user can be used to enable somelocation-based evaluation of the checkout list of the application. Thoselocations for example, could be used to identify a set of candidateproducts possibly weighted by likelihood or a similar metric. In anideal scenario the subproperty of the interaction will corroborate theevaluation from the user application. In other variations, the CVmonitoring system could be used to enable different forms of monitoringalignment verification.

In another variation, a CV monitoring system can be used alongside aprofile prediction system, which is used to use historical profilerecords in evaluating a checkout list or other metrics from the CVmonitoring system. In this variation, determining monitoring alignmentcan include augmenting confidence level in the first evaluation throughcomparison of a historical profile record to a checkout list from the CVmonitoring system. This may be used to compare historical shoppingpatterns to current checkout list. If a low confidence item is notexpected based on historical records of that user (or of theenvironment) then this may be characterized as misalignment. If a userhas previously been involved in one or more checkout sessions withmonitoring misalignment or other forms of exceptions during the checkoutprocess, then that may indicate a high confidence threshold to permitcheckout. For example, users whose behavior triggers manual verificationof items before will have these events recorded in the profile recordand where the profile prediction system may generate an evaluationpredicting a likely misalignment for one or more variety of reasons.This variation will generally involve updating the profile record sothat an updated profile can be used in subsequent visits of a user to anenvironment as shown in FIG. 8.

The systems and methods of the embodiments can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated with apparatusesand networks of the type described above. The computer-readable mediumcan be stored on any suitable computer readable media such as RAMs,ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives,floppy drives, or any suitable device. The computer-executable componentcan be a processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

We claim:
 1. A method for interaction monitoring in a retail environmentcomprising: executing a first monitoring system and thereby generating afirst independent evaluation of customer selection of items from a firstset of sensors; executing a second monitoring system and therebygenerating a second independent evaluation of customer selection ofitems from a second set of sensors; determining, based on the firstindependent evaluation of customer selection of items and the secondindependent evaluation of customer selection of items, monitoringalignment by scoring a degree of agreement between the first independentevaluation and the second independent evaluation of a first customer;and triggering an action in response to the monitoring alignment.
 2. Themethod of claim 1, wherein one of the first monitoring system and thesecond monitoring system is a multi-camera, computer vision monitoringsystem.
 3. The method of claim 2, wherein the first independentevaluation is generated independent from the second independentevaluation, and the second independent evaluation is generatedindependent from the first independent evaluation; and whereindetermining monitoring alignment comprises comparing selected itemsassociated with a first customer in the first independent evaluation toselected items associated with the first customer in the secondindependent evaluation.
 4. The method of claim 2, wherein determiningmonitoring alignment further comprises verifying a set of selected itemsassociated with a first customer of the second independent evaluationmaps to at least a subset of selected items associated with the firstcustomer of the first independent evaluation.
 5. The method of claim 2,wherein determining monitoring alignment comprises evaluating aconditional logic based on properties of the first independentevaluation and the second independent evaluation.
 6. The method of claim2, wherein triggering an action comprises controlling a checkout processin coordination with the monitoring alignment.
 7. The method of claim 6,wherein controlling a checkout process further comprises, whenmonitoring alignment satisfies a condition permitting a checkout processat a checkout system for items of at least one of the first or secondindependent evaluation, and when monitoring alignment does not satisfythe condition, communicating an alert to the checkout system.
 8. Themethod of claim 1, wherein triggering an action comprises, whenmonitoring alignment does not satisfy an alignment condition, generatingworker instructions based on the monitoring alignment and communicatingthe worker instructions to a worker management device.
 9. The method ofclaim 8, wherein the worker instructions direct inspection of at leastone item that has monitoring misalignment.
 10. The method of claim 2,wherein triggering the action can comprise: updating a user profile ofthe first customer, and at subsequent visit, one of the first or secondmonitoring systems detecting the first customer, accessing the userprofile, and augmenting checkout processing based in part on the userprofile.
 11. The method of claim 2, wherein triggering the actioncomprises generating a retail operational instruction that specifies atleast one of: item stocking placement and configuration of at least oneof the first or second monitoring systems.
 12. The method of claim 2,wherein triggering the action comprises updating the computer visionmonitoring system in response to the monitoring alignment.
 13. Themethod of claim 1, wherein the first monitoring system is a computervision monitoring system such that executing the first monitoring systemcomprises collecting image data, applying a CV-based processing, andgenerating the first independent evaluation; wherein the secondmonitoring system is a radio frequency tag monitoring system such thatexecuting the second monitoring system comprises detecting tag presenceand associating tag presence to product presence.
 14. The method ofclaim 13, wherein the radio frequency tag monitoring system is an radiofrequency identification tag monitoring system with an identifier mappedto at least one product; and wherein the second independent evaluationis an at least partial item selection list; and wherein determiningmonitoring alignment comprises comparing the at least partial itemselection list to a selected item list of the computer vision monitoringsystem.
 15. The method of claim 13, wherein the second monitoring systemincludes a radio frequency security tag detector barrier positioned atthe exit of the environment, wherein the second independent evaluationis a binary signal of product presence; and wherein triggering an actiontriggers an alarm of the security tag detector barrier based on themonitoring alignment.
 16. The method of claim 1, wherein the firstmonitoring system is a smart cart monitoring system such that executingthe second monitoring system comprises detecting item-based cartcontents of individual carts in the retail environment; wherein thesecond monitoring system is a computer vision monitoring system thatcomprises an imaging system with partial coverage of the environmentsuch that executing the first monitoring system comprises generating apartial item selection list; and wherein determining monitoringalignment partially verifies the item-based cart contents against thepartial item selection list.
 17. The method of claim 1, wherein thefirst monitoring system is a user application and wherein that executingthe second monitoring system comprises collecting item selection througha user interface and generating an checkout list; wherein the secondmonitoring system is a computer vision monitoring system that comprisesan imaging system with partial coverage of the environment such thatexecuting the second monitoring system comprises generating a partialitem selection list; and wherein determining monitoring alignmentpartially verifies the checkout list contents against the partial itemselection list.
 18. The method of claim 1, wherein the first monitoringsystem is a computer vision monitoring system such that executing thefirst monitoring system comprises collecting image data, applying acomputer vision processing, and generating a checkout list as the firstindependent evaluation; wherein the second monitoring system is aprofile prediction system such that the second evaluation is ahistorical profile record; and wherein determining monitoring alignmentcomprises augmenting confidence level in the first evaluation throughcomparison of a historical profile record to the checkout list.
 19. Anon-transitory computer-readable medium storing computer-readableinstructions that, when executed by one or more computer processors,cause the one or more computer processors to execute: executing a firstmonitoring system and thereby generating a first independent evaluationof customer selection of items from a first set of sensors; executing asecond monitoring system and thereby generating a second independentevaluation of customer selection of items from a second set of sensors;determining, by the decision system and based on the first independentevaluation of customer selection of items and the second independentevaluation of customer selection of items, monitoring alignment byscoring a degree of agreement between the first independent evaluationand the second independent evaluation of a first customer; andtriggering an action in response to the monitoring alignment.
 20. Asystem comprising of: one or more computer-readable mediums storingcomputer-readable instructions that, when executed by the one or morecomputer processors, cause the one or more computer processors toperform execution comprising: executing a first monitoring system andthereby generating a first independent evaluation of customer selectionof items from a first set of sensors; executing a second monitoringsystem and thereby generating a second independent evaluation ofcustomer selection of items from a second set of sensors; determining,by the decision system and based on the first independent evaluation ofcustomer selection of items and the second independent evaluation ofcustomer selection of items, monitoring alignment by scoring a degree ofagreement between the first independent evaluation and the secondindependent evaluation of a first customer; and triggering an action inresponse to the monitoring alignment.